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Abstract 

This  paper  provides  an  overview  of  the  Joint  Concepts  Division  approach  to  evaluation  of  draft 
joint  concepts  through  wargaming.  As  part  of  the  joint  concept  development  process,  in-stride 
concept  evaluation  examines  the  viability  of  draft  joint  concepts  prior  to  final  review.  The 
concept  evaluation  process  aims  to  design  an  evaluation  event  that  supports  a  detailed  analytic 
framework  based  on  clear  objectives.  Analytic  war  games  provide  a  useful  means  for  achieving 
adequate  rigor  at  acceptable  cost.  Preparation  and  execution  of  war  games  must  ensure 
appropriate  data  is  generated  for  analysis.  Concept  evaluation  culminates  with  a  report  of 
findings  and  recommendations  based  on  synthesis  of  the  quantitative  data  and  qualitative 
observations  generated.  The  recommendations  enable  advancement  of  concept  development 
through  critical  examination  and  refinement  of  the  draft  ideas. 
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1.  Purpose  of  this  paper.  This  paper  provides  an  overview  of  the  Joint  Concepts  Division 
approach  to  evaluation1  of  draft  joint  concepts  through  wargaming. 

2.  Introduction.  “The  Joint  Strategic  Planning  System  (JSPS)  is  the  primary  means  by  which 
the  Chairman  of  the  Joint  Chiefs  of  Staff  (CJCS)  carries  out  statutory  responsibilities  assigned  in 
titles  6,  10,  22  and  50  of  the  United  States  Code  (USC).”2  Within  JSPS,  joint  concepts  provide  a 
strategy-based  vision  of  future  joint  force  operations  from  which  required  joint  capabilities  can 
be  determined,  risks  can  be  assessed,  and  informed  decisions  can  be  made.  Joint  concepts 
propose  new  approaches  to  compelling  current  or  future  challenges  for  which  existing 
approaches  and  capabilities  are  insufficient.  Joint  concept  development  is  the  process 
established  in  the  CJCS1  3010.02  series  for  the  formal  generation  of  joint  concepts  from  initial 
identification  through  implementation.  A  critical  step  in  this  process  is  concept  evaluation.  Joint 
concepts  are  formally  evaluated  to  determine  if  they  will  achieve  their  intended  purpose. 

3.  Concept  Evaluation.  CJCSI  3010.02  requires  evaluation  of  concepts  during  the  development 
process  after  the  emerging  ideas  have  matured  but  prior  to  the  first  formal  Joint  Staff  Action 
Package  (JSAP)  review.  This  stage  is  referred  to  as  “in-stride”  concept  evaluation.  Within  the 
established  concept  development  timeline,  in-stride  evaluation  normally  occurs  between  the  0.3 
and  0.5  versions  of  the  draft  concept,  following  a  red  team  review. 

In-stride  concept  evaluation  assesses  the  viability3  of  the  draft  joint  concept,  identifies  significant 
deficiencies  in  the  central  and  supporting  ideas,  and  discovers  potential  risks  and  implications  for 
future  joint  force  development.  The  evaluation  leads  to  refinements  that  improve  the  draft 
concept.  Additionally,  flaws  may  be  identified  which  may  require  significant  changes  to  the 
concept  or  even  a  re-examination  of  the  value  of  proceeding  with  further  development. 

The  process  of  in-stride  concept  evaluation  combines  the  art  of  wargaming  with  analysis  to 
achieve  evidence-based  findings  in  the  assessment  of  the  concept’s  viability.  This  process, 
depicted  in  Figure  1,  begins  with  a  well-defined  purpose  and  is  supported  by  the  following  steps: 
crafting  achievable  and  measurable  objectives;  building  a  framework  for  analysis;  designing  an 
event  that  supports  the  analytic  framework;  preparing  for  and  executing  the  event;  and 
understanding  and  reporting  the  results.  These  steps  are  sequential,  but  in  practice  may  overlap 
while  undergoing  iterative  refinement.  The  remainder  of  this  paper  presents  an  overview  of 
these  concept  evaluation  steps. 


1  Evaluate:  I .  to  determine  or  fix  the  value  of;  2,  to  determine  the  significance,  worth,  or  condition  of  usually  by 
careful  appraisal  and  study.  (Merriam-Webster.com) 

2  CJCSI  3 100.01  B,  Joint  Strategic  Planning  System,  12  December  2008. 

3  Viable:  capable  of  working,  functioning,  or  developing  adequately.  (Merriam-Webster.com) 
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Figure  1.  Concept  Evaluation  Process 

3.1  Purpose  and  Objectives.  The  purpose  of  in-stride  concept  evaluation  is  to  assess  the 
viability  of  the  draft  concept.  It  assesses  viability,  as  opposed  to  validity4  or  effectiveness,5 
because  the  conceptual  ideas  at  this  stage  are  not  fully  developed.  The  evaluation  begins  with 
understanding  the  concept  and  establishing  achievable  and  measurable  objectives.  The 
objectives  address  the  significant  new  conceptual  ideas.  In  addition,  in-stride  concept  evaluation 
identifies  and  assesses  implications,  risks,  and  capabilities  within  the  concept.  Ultimately,  the 
evaluation  should  enable  the  refinement  of  the  draft  concept  and  improve  the  quality  of  future 
drafts.  The  key  to  concept  evaluation  is  that  it  generates  adequate  data  to  address  the  objectives 
of  the  assessment.  This  begins  with  a  well-crafted  analytic  framework. 

3.2  Analytic  Framework.  The  analytic  framework  supports  achieving  the  concept  evaluation 
objectives  by  first  breaking  down  the  conceptual  ideas  into  manageable  elements  for  analysis. 
This  decomposition  results  in  the  identification  of  objective-based  conceptual  issues  and 
corresponding  key  questions  that  form  the  foundation  for  analysis  and  data  collection.  Figure  2 
provides  an  illustration  of  a  notional  analytic  framework. 

Joint  concepts  typically  address  operational  challenges  at  a  high  level.  Evaluation  requires  a  mix 
of  qualitative  and  quantitative  methods  to  analyze  the  ideas  in  the  concept.  In  the  evaluation 
event,  the  concept  is  applied  to  a  problem  set  and  the  data  collected  provides  the  evidence 
required  for  the  assessment.  Survey  data  captures  the  distribution  of  participant  views  on  a 
subject  and  serves  to  bound  acceptable  interpretation  of  war  game  results. 

Evaluation  of  some  joint  concepts  may  require  a  level  of  detail  and  analytic  depth  rooted  in 
quantitative  methods  of  research.  This  may  be  the  case  if  the  concept  is  evaluated  at  a  more 
advanced  level  of  maturity  or  if  there  is  a  need  to  focus  on  a  particular  issue,  joint  function,  or 
capability  within  a  joint  concept.  Regardless  of  the  requirements  for  analysis,  the  analytic 
framework  will  need  to  identify  the  types  of  data  required  to  address  the  objectives.  This 
framework  provides  the  basis  for  concept  evaluation  event  design. 


4  Valid:  2  a:  well-grounded  or  justifiable:  being  at  once  relevant  and  meaningful  b :  logically  correct  3:  appropriate 
to  the  end  in  view  :  EFFECTIVE  (Merriam-Webster.com) 

3  Effective:  1  a :  producing  a  decided,  decisive,  or  desired  effect.  2  :  ready  for  service  or  action  (Merriam- 
Webster.com) 
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Figure  2.  Notional  Analytic  Framework 

3.3  Design.  Concept  evaluation  event  design  facilitates  the  critical  examination  of  the 
conceptual  ideas  in  line  with  the  analytic  framework.  The  goal  is  designing  the  most  efficient 
and  effective  method  to  achieve  this  aim  while  considering  the  limiting  factors  of  time,  funding, 
facilities,  personnel,  and  technology.  There  are  a  number  of  acceptable  ways  to  combine  the  art 
of  design  with  analysis  methodologies  to  evaluate  joint  concepts  within  these  limitations.  This 
paper  addresses  analytic  war  games  as  a  practical  means  of  concept  evaluation.  Analytic  war 
games  provide  a  useful  design  format  that  is  well  suited  for  concept  evaluation  because  of  their 
ability  to  achieve  adequate  rigor  at  an  acceptable  cost. 

3.3.1  Analytic  War  Games.  A  war  game  is  “a  simulated  battle  or  campaign  to  test  military 
concepts  and  usually  conducted  in  conferences  by  officers  acting  as  the  opposing  staffs.”6  An 
analytic  war  game  uses  a  variety  of  operations  research  techniques  in  pursuit  of  the  evaluation 
objectives.  The  game  design  provides  the  structure  and  mechanics  to  generate  useful  and 
adequate  data  to  achieve  the  objectives  as  conveyed  in  the  analytic  framework. 

There  may  be  times  when  concept  evaluation  requires  more  focus  on  a  particular  principle,  joint 
function,  or  capability.  In  this  case,  the  analytic  framework  may  call  for  greater  depth  of 
information  that  can  only  be  achieved  through  more  detailed  baseline  comparisons,  complex 
simulations,  and  quantitative  analysis.  To  a  certain  degree,  analytic  war  game  design  can 
accommodate  this  additional  depth  of  analysis,  but  at  a  higher  cost.  Analytic  war  game  design  is 
highly  adaptable.  Simple  designs  generally  produce  better  results  because  complex  designs  are 
more  likely  to  introduce  noise  into  the  analysis  process. 

3.3.2  War  Game  Design  Requirements.  War  game  design  is  an  art,  but  there  are  several  basic 
requirements  when  designing  war  games  for  concept  evaluation  (Figure  3).  The  war  game  must 


6  Merriam-Webster.com 
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accommodate  two  basic  tests  of  the  concept.  The  first  addresses  the  question,  “Can  the 
conceptual  ideas  be  applied  against  a  likely  challenge  set?”  In  this  case,  a  set  of  experienced 
participants  (blue  cell)  interprets  and  applies  the  conceptual  ideas  against  a  problem  set  that  is 
simulated  through  the  war  game  environment.  This  typically  manifests  in  the  war  game  through 
the  development  of  operational  approaches,  concepts  of  operations  (CONOPS),  or  courses  of 
action.  This  planning  activity  examines  the  ability  of  the  concept  ideas  to  be  understood  and 
applied  to  military  operations.  It  also  requires  the  blue  cell  to  identify  the  capabilities  required 
for  the  approach  and  potential  risks  associated  with  the  approach.  The  second  test  addresses  the 
question,  “How  likely  is  it  that  the  CONOPS  solution  will  succeed?”  This  adjudication  of  the 
planning  activity  requires  that  the  blue  cell  product  be  compared  to  a  competitive  red  cell 
countermove.  This  process  attempts  to  identify  the  strengths  and  weaknesses  of  the  blue  cell 
approach  and  assesses  overall  risk  and  chance  of  success.  These  basic  tests  can  be  applied  to  the 
war  game  format  in  many  different  design  variations.  The  number  and  type  of  cells  employed, 
as  well  as  the  support  structure  of  technology  and  personnel  should  be  optimized  to  provide  the 
most  efficient  and  effective  interactions  required  for  analysis. 


Objectives 

•  Assess  8  refine  ideas  and  solutions  of  draft  joint  concepts 

•  Identify  and  assess  initial  implications  and  risks 
■  Identify  Initial  capability  requirements  and  gaps 
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Key  Questions 

•  How  vjable  are  the  central  and  supporting  ideas  in  enabling  JF  2020  to  overcome  likely  future  challenges? 

•  What  are  the  risks,  implications,  and  potential  mitigations  associated  with  applying  the  central  and  supporting 
ideas  against  likely  future  challenges? 
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Figure  3.  Framework  for  Concept  Evaluation  Analysis  &  Design 

4.  Preparation  and  Execution.  Evaluation  event  preparation  and  execution  address  the  steps 
necessary  to  implement  the  analytic  war  game  design.  The  main  elements  required  to  prepare  for 
and  execute  concept  evaluation  effectively  in  wargaming  are  the  war  game  environment,  war 
game  participation,  concept  training,  and  war  game  products. 
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4.1  War  Game  Environment.  A  critical  element  of  war  game  design  is  the  game  environment. 
The  environment  is  used  to  set  the  stage  for  war  game  execution.  Basic  environment  products 
include  the  scenario,  forces,  and  higher  level  guidance.  For  concept  evaluation,  the  scenario 
should  simply  and  accurately  convey  environmental  factors  and  challenges  that  define  the 
military  problem  portrayed  in  the  concept.  Adversary  and  friendly  capabilities  required  to  assess 
the  concept  must  be  depicted  in  an  appropriate  manner.  The  level  of  detail  of  these  products  is 
determined  by  the  data  requirements  established  in  the  analytic  framework.  It  is  important  for 
these  products  to  properly  set  the  war  game  starting  conditions  with  enough  resolution  for  the 
participant  blue  and  red  cells  to  successfully  accomplish  their  tasks  in  the  allotted  time. 

Preparing  the  environment  products  for  war  game  execution  is  potentially  costly  and  time- 
consuming  and  requires  careful  consideration  in  planning. 

4.2  War  Game  Participation.  Perhaps  the  most  important  element  of  any  war  game  is  the 
participants.  In  concept  evaluation  war  games,  the  quality  of  the  analysis  is  directly  proportional 
to  the  quality  of  the  participants.  The  set  of  participants  should  reflect  the  subject  matter 
expertise  required  to  address  the  particular  focus  of  the  concept,  especially  where  new  principles 
or  capabilities  are  being  considered.  Concept  evaluation  requires  participants  with  relevant  joint 
operational  experience  and  a  mindset  toward  innovative  approaches  to  military  operations.  Joint 
concepts  require  a  broad  set  of  expertise  that  may  include  representatives  from  the  interagency 
and  multinational  communities. 

4.3  Concept  Training.  War  game  participants  need  to  be  familiar  with  and  understand  the 
concept.  Orienting  the  participants  to  the  conceptual  ideas  in  play  is  absolutely  necessary  to 
ensure  meaningful  data  is  generated.  Concept  familiarization  is  best  implemented  before 
participants  arrive  at  the  war  game.  Adding  concept  writing  team  members  to  the  game  cells 
facilitates  understanding  when  concept-based  questions  arise. 

4.4  War  Game  Products.  The  war  game  should  be  designed  and  structured  to  produce  outputs 
that  directly  address  the  analytic  framework  through  the  data  collection  and  analysis  plan.  War 
game  products  include  but  are  not  limited  to:  game  materials  produced  by  the  cells,  the 
observations  collected  during  game  play,  survey  data,  and  out-briefs.  Careful  consideration  must 
be  given  to  the  time  required  to  generate  these  products.  The  use  of  skilled  facilitators  and 
templates  will  help  to  focus  cell  deliberations  on  the  important  analytic  issues.  Typically, 
concept  evaluation  war  games  culminate  with  out-briefs  to  an  executive  audience  from  which 
valuable  insights  and  recommendations  are  gained. 

5.  Analysis,  Synthesis,  and  Reporting.  Analysis  of  a  war  game  involves  breaking  down  the 
raw  game  data  into  analytic  findings.  Synthesis  takes  the  analytic  findings  and  translates  them 
into  meaningful  recommendations  about  the  concept’s  merits,  risks  and  force  development 
implications.  The  underlying  method  employs  several  sources  of  data  to  triangulate  the  viability 
of  a  draft  concept’s  key  elements. 
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During  war  game  execution,  analysts  focus  on  participant  deliberations  as  they  employ  a 
concept-based  operational  approach  in  a  simulated  future  operational  environment.  Analysts’ 
observations  provide  a  qualitative  understanding  of  why  the  game  unfolded  a  certain  way  that 
complements  more  quantitative  data  streams  focused  on  what  happened  during  the  war  game. 
Each  data  stream  contributes  to  the  evidentiary  basis  for  further  examination.  Following  a  war 
game,  the  raw  observations  gathered  by  various  sources  are  transformed  into  evidence-based 
findings  focused  on  the  criteria  established  in  the  analytic  framework.  In  this  context,  findings 
are  conclusions  reached  after  examination  or  investigation  based  on  the  corroboration  of  insights 
from  multiple  sources.  Each  finding  in  a  report  focuses  on  the  elements  of  the  evaluation  criteria 
as  well  as  risks  and  implications  for  future  joint  force  development.  Findings  blend  quantitative 
data  and  qualitative  observations. 

The  evaluation  process  culminates  with  synthesis  presented  in  the  report  as  recommendations. 
Findings  must  be  substantively  significant  to  merit  synthesis  into  a  corresponding 
recommendation.  The  synthesis  process  uses  critical  thinking  to  organize  findings  in  a  way  that 
clearly  addresses  the  objectives  of  the  evaluation.  Ultimately,  the  evaluation  report  provides 
recommendations  that  enable  the  refinement  of  the  draft  concept  and  potentially  improves  the 
quality  of  future  iterations  of  the  concept. 

6.  Conclusion.  As  part  of  the  joint  concept  development  process,  in-stride  concept  evaluation 
examines  the  viability  of  draft  joint  concepts  prior  to  JSAP  review.  The  concept  evaluation 
process  aims  to  design  an  evaluation  that  supports  a  detailed  analytic  framework  based  on  clear 
outcome-oriented  objectives.  Analytic  war  games  provide  a  useful  means  of  achieving  adequate 
rigor  at  acceptable  cost.  Preparation  and  execution  of  war  games  must  ensure  appropriate  data  is 
generated  for  analysis.  Concept  evaluation  culminates  with  a  report  of  findings  and 
recommendations  based  on  synthesis  of  the  quantitative  data  and  qualitative  observations 
generated.  The  recommendations  enable  advancement  of  concept  development  through  critical 
examination  and  refinement  of  the  draft  ideas. 


6 


